Your Transformer May Not be as Powerful as You Expect

Relative Positional Encoding (RPE), which encodes the relative distance between any pair of tokens, is one of the most successful modifications to the original Transformer. As far as we know, theoretical understanding of the RPE-based Transformers is largely unexplored. In this work, we mathematically analyze the power of RPE-based Transformers regarding whether the model is capable of approximating any continuous sequence-to-sequence functions. One may naturally assume the answer is in the affirmative -- RPE-based Transformers are universal function approximators. However, we present a negative result by showing there exist continuous sequence-to-sequence functions that RPE-based Transformers cannot approximate no matter how deep and wide the neural network is. One key reason lies in that most RPEs are placed in the softmax attention that always generates a right stochastic matrix. This restricts the network from capturing positional information in the RPEs and limits its capacity. To overcome the problem and make the model more powerful, we first present sufficient conditions for RPE-based Transformers to achieve universal function approximation. With the theoretical guidance, we develop a novel attention module, called Universal RPE-based (URPE) Attention, which satisfies the conditions. Therefore, the corresponding URPE-based Transformers become universal function approximators. Extensive experiments covering typical architectures and tasks demonstrate that our model is parameter-efficient and can achieve superior performance to strong baselines in a wide range of applications. The code will be made publicly available at https://github.com/lsj2408/URPE.Comment: 22 pages; NeurIPS 2022, Camera Ready Versio

Functional Interpolation for Relative Positions Improves Long Context Transformers

Preventing the performance decay of Transformers on inputs longer than those used for training has been an important challenge in extending the context length of these models. Though the Transformer architecture has fundamentally no limits on the input sequence lengths it can process, the choice of position encoding used during training can limit the performance of these models on longer inputs. We propose a novel functional relative position encoding with progressive interpolation, FIRE, to improve Transformer generalization to longer contexts. We theoretically prove that this can represent some of the popular relative position encodings, such as T5's RPE, Alibi, and Kerple. We next empirically show that FIRE models have better generalization to longer contexts on both zero-shot language modeling and long text benchmarks

Protein phosphatase 2A plays a crucial role in Giardia lamblia differentiation

Author Posting. © The Authors, 2006. This is the author's version of the work. It is posted here by permission of Elsevier B.V. for personal use, not for redistribution. The definitive version was published in Molecular and Biochemical Parasitology 152 (2007): 80-89, doi:10.1016/j.molbiopara.2006.12.001.The ability of Giardia lamblia to undergo two distinct differentiations in response to physiologic stimuli is central to its pathogenesis. The giardial cytoskeleton changes drastically during encystation and excystation. However, the signal transduction pathways mediating these transformations are poorly understood. We tested the hypothesis that PP2A, a highly conserved serine/threonine protein phosphatase, might be important in giardial differentiation. We found that in vegetatively growing trophozoites, gPP2A-C protein localizes to basal bodies/centrosomes, and to cytoskeletal structures unique to Giardia: the ventral disk, and the dense rods of the anterior, posterior-lateral, and caudal flagella. During encystation, gPP2A-C protein disappears from only the anterior flagellar dense rods. During excystation, gPP2A-C localizes to the cyst wall in excysting cysts but is not found in the wall of cysts with emerging excyzoites. Transcriptome and immunoblot analyses indicated that gPP2A-C mRNA and protein are upregulated in mature cysts and during the early stage of excystation that models passage through the host stomach. Stable expression of gPP2A-C antisense RNA did not affect vegetative growth, but strongly inhibited the formation of encystation secretory vesicles (ESV) and water-resistant cysts. Moreover, the few cysts that formed were highly defective in excystation. Thus, gPP2A-C localizes to universal cytoskeletal structures and to structures unique to Giardia. It is also important for encystation and excystation, crucial giardial transformations that entail entry into and exit from dormancy.This work was funded by NIH grants GM61896, AI51687, AI42488, and DK35108. Dr. A.G. McArthur was supported by NIH grant AI51089 and the Marine Biological Laboratory’s Program in Global Infectious Diseases, funded by the Ellison Medical Foundation

Lysyl hydroxylase 2 induces a collagen cross-link switch in tumor stroma

Epithelial tumor metastasis is preceded by an accumulation of collagen cross-links that heighten stromal stiffness and stimulate the invasive properties of tumor cells. However, the biochemical nature of collagen cross-links in cancer is still unclear. Here, we postulated that epithelial tumorigenesis is accompanied by changes in the biochemical type of collagen cross-links. Utilizing resected human lung cancer tissues and a p21CIP1/WAF1-deficient, K-rasG12D-expressing murine metastatic lung cancer model, we showed that, relative to normal lung tissues, tumor stroma contains higher levels of hydroxylysine aldehyde–derived collagen cross-links (HLCCs) and lower levels of lysine aldehyde–derived cross-links (LCCs), which are the predominant types of collagen cross-links in skeletal tissues and soft tissues, respectively. Gain- and loss-of-function studies in tumor cells showed that lysyl hydroxylase 2 (LH2), which hydroxylates telopeptidyl lysine residues on collagen, shifted the tumor stroma toward a high-HLCC, low-LCC state, increased tumor stiffness, and enhanced tumor cell invasion and metastasis. Together, our data indicate that LH2 enhances the metastatic properties of tumor cells and functions as a regulatory switch that controls the relative abundance of biochemically distinct types of collagen cross-links in the tumor stroma

Research on the motion characteristics of a flexible joint-flexible link space manipulator

Research on motion characteristics of the space flexible-joint flexible-link manipulator is investigated. The motion equations of the N degree of freedom (DOF) flexible manipulator are established by means of the Newton-Euler method and finite segment method considering both computational efficiency and accuracy. The genetic algorithm is used to identify the parameters of the torsional stiffness and damping related to the link flexibility. The derived model involves the dynamic factors, such as joint and link flexibility, joint friction and the end lumped mass of the link. Then the joint critical stiffness is adopted to decide whether to consider joint flexibility. After that, spatial motions of a two flexible-joint flexible-link manipulator are performed with different joint stiffness and friction. The results show that the spatial vibration of the flexible manipulator can be reduced by increasing both the joint stiffness and friction, and the vibration can be effectively suppressed when the joint stiffness is greater than the critical value. Meanwhile, the validity of the presented model is verified. It lays the foundation for the reliability analysis and controller design of the flexible-joint flexible-link space manipulator

MYOLO: A Lightweight Fresh Shiitake Mushroom Detection Model Based on YOLOv3

Fruit and vegetable inspection aids robotic harvesting in modern agricultural production. For rapid and accurate detection of fresh shiitake mushrooms, picking robots must overcome the complex conditions of the growing environment, diverse morphology, dense shading, and changing field of view. The current work focuses on improving inspection accuracy at the expense of timeliness. This paper proposes a lightweight shiitake mushroom detection model called Mushroom You Only Look Once (MYOLO) based on You Only Look Once (YOLO) v3. To reduce the complexity of the network structure and computation and improve real-time detection, a lightweight GhostNet16 was built instead of DarkNet53 as the backbone network. Spatial pyramid pooling was introduced at the end of the backbone network to achieve multiscale local feature fusion and improve the detection accuracy. Furthermore, a neck network called shuffle adaptive spatial feature pyramid network (ASA-FPN) was designed to improve fresh shiitake mushroom detection, including that of densely shaded mushrooms, as well as the localization accuracy. Finally, the Complete Intersection over Union (CIoU) loss function was used to optimize the model and improve its convergence efficiency. MYOLO achieved a mean average precision (mAP) of 97.03%, 29.8M parameters, and a detection speed of 19.78 ms, showing excellent timeliness and detectability with a 2.04% higher mAP and 2.08 times fewer parameters than the original model. Thus, it provides an important theoretical basis for automatic picking of fresh shiitake mushrooms

Research on Instance Segmentation Algorithm of Greenhouse Sweet Pepper Detection Based on Improved Mask RCNN

The fruit quality and yield of sweet peppers can be effectively improved by accurately and efficiently controlling the growth conditions and taking timely corresponding measures to manage the planting process dynamically. The use of deep-learning-based image recognition technology to segment sweet pepper instances accurately is an important means of achieving the above goals. However, the accuracy of the existing instance segmentation algorithms is seriously affected by complex scenes such as changes in ambient light and shade, similarity between the pepper color and background, overlap, and leaf occlusion. Therefore, this paper proposes an instance segmentation algorithm that integrates the Swin Transformer attention mechanism into the backbone network of a Mask region-based convolutional neural network (Mask RCNN) to enhance the feature extraction ability of the algorithm. In addition, UNet3+ is used to improve the mask head and segmentation quality of the mask. The experimental results show that the proposed algorithm can effectively segment different categories of sweet peppers under conditions of extreme light, sweet pepper overlap, and leaf occlusion. The detection AP, AR, segmentation AP, and F1 score were 98.1%, 99.4%, 94.8%, and 98.8%, respectively. The average FPS value was 5, which can be satisfied with the requirement of dynamic monitoring of the growth status of sweet peppers. These findings provide important theoretical support for the intelligent management of greenhouse crops

Citrus Tree Crown Segmentation of Orchard Spraying Robot Based on RGB-D Image and Improved Mask R-CNN

Orchard spraying robots must visually obtain citrus tree crown growth information to meet the variable growth-stage-based spraying requirements. However, the complex environments and growth characteristics of fruit trees affect the accuracy of crown segmentation. Therefore, we propose a feature-map-based squeeze-and-excitation UNet++ (MSEU) region-based convolutional neural network (R-CNN) citrus tree crown segmentation method that intakes red–green–blue-depth (RGB-D) images that are pixel aligned and visual distance-adjusted to eliminate noise. Our MSEU R-CNN achieves accurate crown segmentation using squeeze-and-excitation (SE) and UNet++. To fully fuse the feature map information, the SE block correlates image features and recalibrates their channel weights, and the UNet++ semantic segmentation branch replaces the original mask structure to maximize the interconnectivity between feature layers, achieving a near-real time detection speed of 5 fps. Its bounding box (bbox) and segmentation (seg) AP50 scores are 96.6 and 96.2%, respectively, and the bbox average recall and F1-score are 73.0 and 69.4%, which are 3.4, 2.4, 4.9, and 3.5% higher than the original model, respectively. Compared with bbox instant segmentation (BoxInst) and conditional convolutional frameworks (CondInst), the MSEU R-CNN provides better seg accuracy and speed than the previous-best Mask R-CNN. These results provide the means to accurately employ autonomous spraying robots

Extension Cloud Model and Grey Clustering Evaluation of Enterprise Safety Management System: Based on COWA-CRITIC Combination Weighting

In order to address the issues of unclear risk grading control, lack of safety management, and hidden danger investigation and management processes, this paper used a mining enterprise as the backdrop for an engineering example. The “evaluation model of the overall construction level of the enterprise safety management system” is constructed from four aspects: “preliminary infrastructure”, “risk grading and control”, “hidden danger investigation and management processes”, and “Post-support work”. The safety evaluation level is divided into five levels, and the evaluation model is combined weighted by using the combined ordered weighted averaging (COWA) algorithm and the criteria importance through intercriteria correlation (CRITIC) method. In addition, the cloud model, the extension cloud model, and the grey clustering evaluation method are used for a thorough evaluation. Finally, the enterprise safety management system’s overall construction level is determined to be good. In order to effectively strengthen the enterprise safety management capability and prevent the occurrence of production safety accidents, this study provides a practical and thorough evaluation method for the evaluation of the enterprise safety management system. This method makes it easier to identify system weaknesses and provides a safety guarantee for the sustainable development of enterprises

Expression of wild-type PtrIAA14.1, a poplar Aux/IAA gene causes morphological changes in Arabidopsis

Aux/IAA proteins are transcriptional repressors that control auxin signaling by interacting with Auxin Response Factors (ARFs). So far all of the identified Aux/IAA mutants with auxin-related phenotypes in Arabidopsis and rice (Oryza sativa) are dominant gain-of-function mutants, with mutantions in Domain II that affected stability of the corresponding Aux/IAA proteins. On the other hand, morphological changes were observed in knock-down mutants of Aux/IAA genes in tomato (Solanum lycopersicum), suggesting that functions of Aux/IAA proteins may be specific for certain plant species. We report here the characterization of PtrIAA14.1, a poplar (Populus trichocarpa) homologue of IAA7. Bioinformatics analysis showed that PtrIAA14.1 is a classic Aux/IAA protein. It contains four conserved domains with the repressor motif in Domain I, the degron in Domain II, and the conserved amino acid signatures for protein-protein interactions in Domain III and Domain IV. Protoplast transfection assays showed that PtrIAA14.1 is localized in nucleus. It is unable in the presence of auxin, and it represses auxin response reporter gene expression. Expression of wild type PtrIAA14.1 in Arabidopsis resulted in auxin-related phenotypes including down-curling leaves, semi-draft with increased number of branches, and greatly reduced fertility, but expression of the Arabidopsis Aux/IAA genes tested remain largely unchanged in the transgenic plants. Protein-protein interaction assays in yeast and protoplasts showed that PtrIAA14.1 interacted with ARF5, but not other ARFs. Consistent with this observation, vascular patterning was altered in the transgenic plants, and the expression of AtHB8 (Arabidopsis thaliana Homeobox Gene 8) was reduced in transgenic plants